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ABSTRACT 


Biomass is expected to play an increasingly significant role in the ‘greening’ of energy supply. Neverthe- 
less, concerns are rising about the sustainability of large-scale energy crop production. Impacts must be 
assessed carefully before deciding whether and how this industry should be developed, and what tech- 
nologies, policies and investment strategies should be pursued. There is need for a comprehensive and 
reliable sustainability assessment tool to evaluate the environmental, social and economic performance 
of biomass energy production. This paper paves the way for such a tool by analysing and comparing 
the performance and applicability of a selection of existing tools that are potentially useful for sustain- 
ability assessment of bioenergy systems. The selected tools are: Criteria And Indicators (C&I), Life Cycle 
Assessment (LCA), Environmental Impact Assessment (EIA), Cost Benefit Analysis (CBA), Exergy Analysis 
(EA) and System Perturbation Analysis (SPA). To evaluate the tools, a framework was constructed that 
consists of four evaluation levels: sustainability issues, tool attributes, model structure, area of applica- 
tion. The tools were then evaluated using literature data and with the help of a Delphi panel of experts. 
Finally, a statistical analysis was performed on the resulting data matrix to detect significant differences 
between tools. It becomes clear that none of the selected tools is able to perform a comprehensive sus- 
tainability assessment of bioenergy systems. Every tool has its particular advantages and disadvantages, 
which means that trade-offs are inevitable and a balance must be found between scientific accuracy and 
pragmatic decision making. A good definition of the assessment objective is therefore crucial. It seems 
an interesting option to create a toolbox that combines procedural parts of C&I and EIA, supplemented 
with calculation algorithms of LCA and CBA for respectively environmental and economic sustainability 
indicators. Nevertheless, this would require a more comprehensive interdisciplinary approach to align 
the different tool characteristics and focuses. 
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1. Introduction 
1.1. Current state of sustainability assessment 


The concept of sustainability has evolved considerably since it 
was mentioned for the first time in the forestry sector of the 18th 
century [1-8]. Nowadays, the standard definition of sustainable 
development provided by the Brundtland Commission “to ensure 
that it meets the needs of the present without compromising the 
ability of future generations to meet their own needs” [9] is gener- 
ally recognised as a standard and a starting point for most who set 
out to define the concept. The concept is often depicted schemat- 
ically using three overlapping circles for the target dimensions of 
environment, economy and society, to which might be added an 
institutional dimension [10]. More detailed discussions of differ- 
ent concepts of sustainability can be found in various publications 
[11-13]. 

For the transition towards a more sustainable society, com- 
prehensive but efficient and reliable assessment tools are needed. 
Devuyst et al. [14] define sustainability assessment as “a tool that 
can help decision-makers and policy-makers decide which actions 
they should or should not take in an attempt to make society more 
sustainable.” Following Ness et al. [15] the purpose of sustainabil- 
ity assessment is to provide decision-makers with an evaluation 
of global to local integrated nature-society systems in short and 
long term perspectives in order to assist them to determine which 
actions should or should not be taken in an attempt to make society 
more sustainable. Based on the four dimensions of sustainabil- 
ity, several comprehensive frameworks for integrated assessment 
have been developed [16-18]. In 1996, a group of researchers have 
established a set of practical guidelines in the form of overarch- 
ing principles that provide a link between theory and practice for 
the whole assessment process from system design and identifi- 
cation of indicators, through field measurements and compilation 
to interpretation and communication of results, which are known 
as the “Bellagio Principles” [19]. In the last decade, a number of 
concepts have been developed that aim at upgrading the opera- 
tionalisation of the vision of sustainability. This resulted in a diverse 
field of approaches that claim that they can be used for assessing 
sustainability in many different sectors [15,18-27]. Sustainability 
assessment has increasingly become associated with the family of 
impact assessment tools such as Life Cycle Assessment, Ecological 
Footprint, Environmental Impact Assessment and Strategic Envi- 
ronmental Impact Assessment [8,28]. 


1.2. Problem statement, aim and approach 


In this research, the assessment focus is on the bioenergy sector. 
Worldwide, the potential of biomass to contribute to reductions 
in greenhouse gas (GHG) emissions, to improved energy security 
and to rural diversification and development is recognised [16]. 
Several bioenergy policies such as the Brazilian Biofuel Law, the 
Indian Biofuel Policy, the U.S. Renewable Fuel Standard, and Euro- 


pean policies such as the Renewable Energy Directive, Strategy 
on Biofuels and the Biomass Action Plans push the implementa- 
tion of energy derived from biomass in order to reduce fossil fuel 
dependency or to meet greenhouse gas reduction targets. These 
policies are a source of increasing concern due to questions over 
the sustainability of large-scale bioenergy crop production. Poten- 
tial negative impacts include direct and indirect land use change 
[29-32] and biodiversity loss [33,34], availability of water resources 
[35-38], rising agricultural commodity prices and threats to food 
security [39-44]. These risks have to be weighed against potential 
benefits such as improved greenhouse gas balance, employment 
and income generation, rural development, conversion of conven- 
tional industries and increased security of energy supply [45-47]. 
Therefore, the environmental, social and economic impacts of 
bioenergy development needs to be assessed carefully before 
deciding whether and how this industry should be developed, and 
what technologies, policies and investment strategies should be 
pursued. In this context, some governments and institutions started 
developing sustainability tools and standards to evaluate the envi- 
ronmental, social and economic performance of biomass energy 
production. At the same time, companies are willing to integrate 
more sustainable strategies in their energy management and are 
looking for practical and reliable sustainability assessment tools 
to assess different energy system possibilities. However, the high 
variability in biomass sources, conversion technologies and con- 
texts complicate such assessments. A variety of assessment tools 
for sustainability evaluation of production systems has been devel- 
oped in the past, each showing a different procedure and field of 
application [48]. 

The general aim of this paper is to contribute to the development 
of a comprehensive, yet practical and reliable tool for sustainability 
assessment of bioenergy systems. Since many tools currently exist, 
there is more of a need to highlight complementarities or possibili- 
ties for integration rather than to generate new tools. Therefore, the 
specific goal of this paper was to make a comparative evaluation of 
a selection of existing tools to which degree these tools are able 
to incorporate the different dimensions of sustainability of bioen- 
ergy systems. The research approach is presented in Fig. 1. First, an 
evaluation framework is constructed in which the tools are evalu- 
ated. In a next step, the evaluation of the selected tools is made. The 
resulting data matrix is then statistically analysed to detect signif- 
icant differences between tools and as a conclusion, strengths and 
weaknesses of the tools are highlighted and possibilities for their 
integration in a more comprehensive tool are made. 


2. Materials and methods 
2.1. Selection of existing assessment methods 


A wide range of sustainability assessment methods has been 
developed in recent years hence some initial selection was needed. 
Our selection strategy was twofold. On the one hand we chose some 
well-known and commonly used but conceptually very different 
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STEP 1: 


Construction of an 
evaluation framework 


STEP 2: STEP 3: 
Evaluation of Overview of strengths 
assessment tools and weaknesses 


Levels of evaluation: 


Tool attributes 


Model structure 
Area of application 


Fig. 1. Flow chart of the tool evaluation process. 


tools for sustainability assessment: Criteria And Indicators (C&I), 
Life Cycle Analysis (LCA), Environmental Impact Assessment (EIA) 
and Cost Benefit Analysis (CBA). On the other hand we added two 
rather uncommon but conceptually promising tools. Exergy Anal- 
ysis (EA) is a method in thermodynamics able to account material 
and energy flows all in the same unit. System Perturbation Analysis 
(SPA) is a tool that performs an accountancy evaluation after per- 
turbation of a system (e.g. increase in bioenergy production) and 
compensation of the disturbances. SPA was specifically designed for 
the Belgian system. The result of the selection is a mixture of tools 
with varying scopes and assessment strategies, focussing on dif- 
ferent systems (products, services, projects and/or economies) and 
various impacts. A short introduction to each tool is given in Box 1 


Box 1 

Criteria and Indicators (C&I) can be used for a wide range of 
applications such as eco-certification at the management unit 
level [49], policy evaluation at the regional or national level 
[50,51], or as a generic assessment tool for specific environ- 
mental, social, economic or institutional sustainability issues. 
The use of C&I is a common way to describe and monitor com- 
plex systems, and to provide information to decision makers 
and the public. Some well-known sustainability criteria initia- 
tives in the field of bioenergy [52-54] are the criteria of the 
Cramer Commission [17,55], the Renewable Transport Fuels 
Obligation [56-58], the European Renewable Energy Directive 
[59], the Round Table for Sustainable Palm Oil [60,61], the 
Round Table on Responsible Soy [62], the Basel Criteria for 
Responsible Soy Production [63], the Roundtable on Sustain- 
able Biofuels [64-66], the Forest Stewardship Council FSC [67] 
and the Programme for the Endorsement of Forest Certification 
PEFC [68,69]. 

A well-established method is Life Cycle Assessment (LCA). 
LCA has been used in varying forms over the past 35 years 
to evaluate the environmental impacts of a product or a ser- 
vice throughout its life cycle. It is an approach that analyses 
real and potential pressure that a product has on the envi- 
ronment during its whole life cycle from cradle to grave, 
including raw material acquisition, production process, use, 
and disposal of the product [70]. The International Standards 
Organisation (ISO) has established guidelines and principles 
for LCA that have been further interpreted and developed 
by many authors [25,71-76]. LCA results provide information 
for decisions regarding product development and ecodesign, 
production system improvements, and product choice at the 
consumer level. Numerous Life-cycle Assessments have been 
performed in the bioenergy field [77-88], as well as for a mul- 
titude of other product and service areas. 

Environmental Impact Assessment (EIA) has been used since 
1960s for evaluating potential environmental impacts — con- 
sidering natural, social and economic issues — that a proposed 
project may have, with the aim to reduce its negative 


Sustainability aspects 


Kruskal-Wallis 
analysis of variance 


+ 
Tool profiles 


Literature review 


effects [89]. The purpose of the assessment is to decide 
whether to proceed with the project and to identify impact 
mitigating measures. The International Association for Impact 
Assessment (IAIA) defines an Environmental Impact Assess- 
ment as the process of identifying, predicting, evaluating and 
mitigating the biophysical, social, and other relevant effects of 
development proposals prior to major decisions being taken 
and commitments made [90]. In the EU, a directive that made 
EIA compulsory for proposed public and private projects (e.g. 
construction projects) that are likely to have environmental 
impacts was introduced in 1985 [91,92]. EIA has also been intro- 
duced in the legislation in many other countries [93]. Due to 
such legal requirements, there are strict guidelines for the EIA 
process in the EU and other countries [89]. In literature, no EIA 
studies were found for bioenergy projects in particular. 

Cost Benefit Analysis (CBA) is an applied welfare economics 
tool with roots reaching back to the early 20th century. It is 
used for evaluating public or private investment proposals by 
weighing the costs of the project against the expected ben- 
efits. In the realm of sustainability assessment, CBA can be 
an effective tool for weighing the social costs and benefits of 
different alternatives in connection with e.g. energy and trans- 
ports [94]. It is this aspect of measuring expected benefits, or 
placing monetary units on the benefits, e.g. biodiversity, that 
is often a challenge within CBA [95]. A number of CBA studies 
were found regarding bioenergy projects [96-100]. 

Exergy Analysis (EA) is a powerful thermodynamic account- 
ing technique for assessing and improving the efficiency of 
processes, devices and systems, as well as for enhancing envi- 
ronmental and economic performance. As a multidisciplinary 
concept, exergy applications are observed in various fields, 
including mechanical and chemical engineering as well as eco- 
nomics, management, physics and biology. Consequently, EA 
is used increasingly by industries and governments through- 
out the world, particularly with the aim of improving energy 
sustainability. In depth explanations of the concept and appli- 
cation of exergy analyses are delivered by many scientist 
[101-109], and examples of exergy assessment related to 
the bioenergy field can be found in [110-112,102,113,114]. 
A new method called System Perturbation Analysis (SPA) was 
recently developed by the Free University of Brussels (VUB) 
[115]. SPA differs from a conventional Life Cycle Analysis (LCA) 
mainly because it looks to alternations in resource balances 
(so-called perturbations) within a specific geographical system 
and the resulting effects, rather than comparing life-cycle tra- 
jectories. Therefore SPA is able to identify the best usage of 
limited resources such as hectares, wood waste or imports, 
in terms of fossil energy savings or GHG emissions within a 
given system. Comparative results of such a System Pertur- 
bation Analysis for the use of wood for transport, heat and 
power applications are presented in the study of Bram et al. 
[115] and results for biomass in a natural gas combined cycle 
power plant can be found in the study of Delattin et al. [116]. 
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2.2. Step 1: construction of an evaluation framework for 
sustainability assessment tools 


The selected tools represent different ways of looking at a given 
assessment challenge and they convey different information to the 
decision process. Each tool collects, structures and evaluates infor- 
mation about specific issues and presents it in a specific way. Which 
specific tool will be used in a specific situation depends on the gen- 
eral conceptual view of the decision maker, the context for which to 
take a decision and the specific objectives of the assessment [117]. 
In order to make a clear evaluation of the selected tools and to com- 
pare their characteristics in terms of objectives, focus and approach, 
an evaluation framework is constructed. The definition of an eval- 
uation framework depends on the specific aim of a study, which 
in our case is the development of a comprehensive yet practical 
and reliable method for sustainability assessment of bioenergy sys- 
tems. ‘Comprehensive’ means that as many environmental, social, 
economic and institutional sustainability issues as possible should 
be considered in the tool. A ‘practical’ tool makes demands on the 
effective use of the tool and is related to the amount of data that 
are needed, which procedure is applied, how the results are repro- 
duced, and in which context the tool can be used. ‘Reliable’ requires 
a transparent and science based model structure that takes all 
inputs, outputs, interactions and interdependencies at each stage 
of the bioenergy system into account [118]. All these requirements 
which determine whether a tool is suitable or not are addressed in 
the evaluation framework, using four levels of comparison: (i) sus- 
tainability issues; (ii) tool attributes; (iii) model structure; and (iv) 
area of application. The first two levels could be analysed numer- 
ically by a Delphi panel of experts, whereas the last two levels, 
which are descriptive, are evaluated based on literature review. 
These four levels of comparison of the evaluation framework are 
further elaborated in the following paragraphs. 


2.2.1. Issues of sustainability 

An important part in determining the ability of a tool to 
serve as a sustainability assessment tool for bioenergy sys- 
tems is to check which sustainability issues are addressed 
by the tool. For this aim, a comprehensive list of relevant 
sustainability issues was constructed, based on sustainability 
requirements for bioenergy systems mentioned in literature, 
e.g. [2,17,52,56,57,59,62,63,119-122]. The resulting list, organised 
along the four dimensions of sustainability, is shown in Table 1. It 
was chosen not to go in too much detail to keep the list operational. 
A few indicators are added as an example. 


2.2.2. Tool attributes 

Assessment tools ideally have a number of characteristics which 
facilitate their application with respect to the use of data, the fol- 
lowed methodological procedure, the reproduction of the results, 
and finally, the use of the tool in general. In this second level of the 
evaluation framework, it is tried to consider a large range of facili- 
tating attributes that a sustainability assessment tool for bioenergy 
systems might have. A list of attributes was made using informa- 
tion about tool attributes, found in various literature resources 
[1,48,122], complemented with own expertise. The result can be 
found in Table 2, in which attributes are given with respect to data 
used in the tool, tool procedure, tool results, and use of the tool. 


2.2.3. Overall model structure 

Literature review of assessment frameworks, tools and compar- 
ison studies has demonstrated that tools can be categorised based 
on several features with regard to the overall model structure e.g. 
[11,15,77,95,123-126]. Of all the features encountered in literature, 
those that seemed relevant for sustainability assessment of bioen- 
ergy systems were withheld for the third level of the evaluation 


Table 1 
Set of sustainability issues for bioenergy systems (examples of indicators between 
brackets). 


Sustainability issues and some of their indicators 

Environmental dimension 
Air quality (emissions of GHG, nitrogen, POP’s) 
Soil quality (nutrient balances, accumulation of heavy metals, soil 
compaction and erosion) 
Land use (amount of land used, leakage effects) 
Water quality (water pollution, eutrophication, pesticide residues) 
Use of ground and surface water 
Biodiversity and preservation of sensitive ecosystems (use of GMO’s/exotic 
species, preservation of habitats and high conservation value forests, 
avoidance of pollution of neighbouring ecosystems) 
Deforestation 
Desertification and drought 
Landscape view (landscape variation, conservation of typical elements) 
Use of agrochemicals 
Waste management (minimization, sorting, disposal, recycling) 
Use of energy (energy balance) 
Use of fossil resources 

Social dimension 
Labour conditions (freedom of association, collective bargaining, minimum 
wages, discrimination) 
Protection of human safety and health (protection of human health, safe and 
healthy work environment) 
Protection of human rights (rights of children, women, indigenous people, 
discrimination) 
Access to resources ensuring adequate quality of life (access to potable 
water, sanitary facilities, adequate housing, education and training, 
transportation, and health services) 
Food and energy supply safety (quantity and quality of food available, 
energy supply) 
Capacity building (building and use of local labour and skills) 
Democratic participation (stakeholder involvement) 
Protection of property rights and user ownership (land tenure conflicts, 
equitable land ownership) 
Fair trade conditions (transparency and accountability of negotiations, fair 
and equal remuneration) 
Social acceptance (acceptance of the business by producer, consumer and 
local population) 

Economic dimension 
Viability of the business (net present value, minimisation of costs, adequate 
funding) 
Long term perspective (long-term commitments, contracts and 
management plans) 
Reliability of resources (security of supply) 
Reliability of the used technology (adequately proven technology) 
Risk minimisation 
Strength and diversification of local economy 
Reliability of energy 
No blocking of other desirable developments 

Institutional dimension 
In respect with local, national and international policy and legislation 
Institutional capacity 
Institutional well-being 
Environmental management 


framework of this study. After a short explanation of a few general 
tool aspects such as tool description, tool concept, object of analy- 
sis, type of comparison, degree of standardisation or harmonisation, 
and frequency of use, some more specific methodological aspects 
are addressed: time frame, spatial scale, concept focus, quantita- 
tive or qualitative character, reproduction of results, assignment 
of weight scores, and sensitivity and uncertainty analysis. A short 
description of the different aspects follows. 

The tool description describes the purpose of the tool, the reason- 
ing behind the development of the tool. The tool concept gives an 
overview of the methodology followed or the different steps during 
application of the tool. The object of analysis specifies which object 
is being analysed by the tool, e.g. a product, a function or an econ- 
omy. Most tools include some type of comparison, either between 
different alternatives, or within a studied system, or against a ref- 
erence. It differs from tool to tool and is often related to the object 
analysed in the decision process [123]. Also the degree of standardis- 
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Table 2 
List of attributes used to describe the characteristics of the evaluated tools. 
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Tool attribute 


Attribute explanation 


Data 
Availability 


Need 


Type 
Integration 


Correctness 
Procedure 
Flexibility 


Coverage 
Objectivity 


Transparency 

Validation 

Time scale 

State-of-the-art 
Results 

Univocity 


Simplicity 


Type 
Clarity 


Communication 
Use 

Flexibility 

Accessibility 


Application cost 
Application field 


Data needed to apply the tool are readily 
available 

Balanced amount of data needed to apply the 
tool 

Preference for quantitative (metric) data input 
Ability to combine descriptive and quantitative 
data 

Correctness of data used 


Model procedure is dynamic, can easily be 
adapted or complemented 

Degree to which multiple sustainability issues 
of the four dimensions are integrated 

Model is not susceptible to the subjectivity of 
the executor 

Transparency of the evaluation procedure 
Ability to validate evaluation results 

Ability to process dynamic data 

Degree to which the tool is operational 


Results are consistent, no contradictions in the 
results 

Results are possible to understand by 
stakeholders, no expert knowledge required 
Results are preferably quantitative 

Results are compact and clear, no long lists or 
extensive reports 

Results are represented graphically, easy to 
interpret 


Ability to integrate inputs of the stakeholders 
Possibility to interpret results without expert 
intervention 

Low cost to execute the evaluation task 

Ability to execute evaluation tasks on different 


levels (micro, macro, ...) 

Low time demand to execute the evaluation 
task 

The model is easy to use 

Availability of experts to render assistance 


Time consumption 


Handiness 
Assistance 


ation or harmonisation is an important aspect, even if many aspects 
are under continuous development [95]. If different users apply 
the tool in separate ways, evolution will make it harder to keep the 
tool homogenous in the future and to compare or share data and 
results. Another aspect is the frequency of use, which gives informa- 
tion about how often the tool is used, and sometimes also where it 
is used. The time frame both indicates the time perspective applied 
to the object analysed which can be retrospective or prospective, as 
well as the time horizon of the object itself and the related impacts. 
Approaches mainly used in the context of accounting, tracking and 
monitoring on-going (e.g. year on year) progress can be called retro- 
spective, while those used to predict future situations can be called 
prospective [123]. The time horizon indicates whether the tool is a 
snapshot in time, or whether it considers the whole life time of a 
product, project or process and/or their impacts. The spatial scale is 
related to the spatial boundaries and horizon of the object analysed. 
Spatial boundaries can be the boundaries of a country or town, but 
also boundaries surrounding a section in a product chain, or the 
boundary between nature and the human system [95]. The spatial 
horizon may include a single site, many sites, or no sites e.g. in case 
of product design. The spatial scale can therefore be categorised as 
site specific focus and non- or multi-site focus [123]. The concept 
focus indicates whether a tool emphasises procedure or focuses on 
the construction of a defined type of computational algorithm. Pro- 
cedure aims at describing the best way to reach a decision, whereas 
algorithms are aimed at finding the better decision [123]. Tools can 
have a purely quantitative character or rather qualitative, or both 


quantitative and qualitative. Under reproduction of results, infor- 
mation is given about how the results are presented, whether they 
are listed with different units, or aggregated in one single outcome. 
Another aspect that is regarded, is the possibility within the tool to 
assign weight scores and how this is performed. In a final part of the 
evaluation, information is gathered on the need to conduct and the 
integration of a sensitivity and/or uncertainty analysis in the tool. 


2.2.4. Application of the tools 

In the fourth level of the evaluation framework the purpose is 
to focus on the application of the tools. For that purpose, various 
aspects can be taken in consideration such as the overall pur- 
pose of the tool (communication, decision-support or learning), the 
user category (companies, government/authorities, NGOs) or the 
object of analysis (product, function or economy) [123]. A study 
conducted by Ness et al. [15] performs a categorisation of assess- 
ment tools based on the temporal focus of the tool along with the 
object of focus of the tool. The temporal focus can be either retro- 
spective (indicators/indices), prospective (integrated assessment) 
or both (product-related assessment). The object of focus of the 
tools is either spatial, referring to a proposed change in policy (indi- 
cator/indices and integrated assessment), or at the product level 
(product-related assessment). For this paper, the same exercise was 
performed as well as an evaluation based on the development stage 
of the tool and the frequency of application. 


2.3. Step 2: evaluation of existing assessment tools 


In the second step of the study, the newly constructed evaluation 
framework is used to evaluate the six selected assessment tools. 

In the first two evaluation levels (sustainability issues and tool 
attributes), the research was carried out with the use of a Delphi 
panel of experts. The Delphi method is a popular technique for fore- 
casting and an aid in decision-making based on the opinions of 
experts, which has been in existence for over half a century [127]. 
The aim of the study was to gather a group of scientist with exper- 
tise in the sustainability assessment field, including at least one 
expert for each of the selected assessment tools. Unfortunately, no 
experts could be included for exergy assessment. As a consequence, 
the Exergy Analysis tool was only evaluated in the third and fourth 
level of the evaluation framework, for which no Delphi expertise 
was required. The experts first received documentation about the 
aim and planning of the study by e-mail. Next, an interactive meet- 
ing was organised, during which grades were assigned to each of 
the tools for each of the sustainability issues and tool attributes 
(level 1 and 2 of the evaluation framework). Experts were able to 
jointly discuss and exchange thoughts during the assignment. The 
grades were ordinal numbers from one to five by which one stands 
for the lowest score and five for the highest. 

The material used to evaluate the assessment tools in level 3 and 
4 of the evaluation framework (overall model structure and tool 
application) consisted of scientific papers describing the method- 
ology of the tools, literature related to specific application of each 
of the assessment tools in the field, and practical examples. 


2.4. Step 3: overview of strengths and weaknesses 


The evaluation of the first and second level of the evaluation 
framework, carried out by the Delphi panel of experts, resulted in 
two data matrices consisting of ordinal scores, giving an overview 
of the strengths and weaknesses of the different tools. In order 
to detect significant differences between tools, the data matrices 
were Statistically analysed. Because of the ordinal character of the 
data, a Kruskal-Wallis analysis of variance was performed. The 
Kruskal-Wallis test is a non-parametric method for testing equal- 
ity of population medians amongst groups. However, in this study 
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we had only one observation for every evaluator variable sepa- 
rately, so no medians could be generated. Therefore, significant 
differences (« =0.05) between tools were evaluated at the subgroup 
level. In the data matrix of the sustainability issues, the subgroups 
are the environmental, social, economic and institutional dimen- 
sion. In the data matrix of the tool attributes, the subgroups are 
data, procedure, results and use. 

The data gathered about the overall model structure in the third 
level of the evaluation framework are reproduced in the form of tool 
profiles, which enables rapid comparison of the different assess- 
ment tools. 


3. Results and discussion 


In comparing sustainability assessment tools, many different 
tool characteristics and aspects are involved. Therefore, an eval- 
uation framework was constructed, distinguishing four levels of 
evaluation: issues of sustainability, tool attributes, overall model 
structure and tool application, in which tools were evaluated. While 
recognising the importance of such a comparative study, it is impor- 
tant to note that the results are not always very straightforward 
because of many reasons. Although the Delphi method is consid- 
ered as a very valuable way to provide information, it cannot be 
avoided that expert valuations are prone to some degree of subjec- 
tivity. Furthermore, most of the considered tools are flexible, are 
developing over time, and not always well defined. Available liter- 
ature might be outdated or not complete. At last, tools can be devel- 
oped for many different reasons, and therefore do not always serve 
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for a straightforward comparison. It is important to keep these 
issues in mind while examining the results of this study. In the fol- 
lowing paragraphs, results are discussed for each evaluation level. 


3.1. Issues of sustainability 


As sustainability is a complex and multidimensional sub- 
ject, integrated sustainability assessment requires a multicriterial 
approach. For this study, important sustainability issues were listed 
for the bioenergy field and tools were valuated with the use of a 
Delphi panel of experts. The lowest score of 1 was given in case the 
tool did not address the aspect at all; a value of 2 when the aspect 
was only addressed to a very limited extend; a value of 3 when the 
tool addresses the aspect with a certain methodology but only par- 
tially; a value of 4 in case the aspect is very well addressed, but still 
could be improved significantly; and a value of 5 in case the aspect 
involved is fully covered (Table 3). 

The results give a clear overview of the sustainability areas that 
are covered by the different tools. It can be noticed that most 
of the tools are environmentally focused. Except for a few minor 
issues, LCA, EIA and SPA only take the environmental dimension 
into account, whereas CBA is rather an economic tool incorporat- 
ing some environmental issues. For the environmental dimension, 
there is a broad agreement in the scientific community that LCA 
is one of the strongest players in the assessment field [70,83], 
although C&I and EIA also show good performances. The scope and 
methodologies of EIA have evolved greatly over the past decade. 
Advances include consideration of biodiversity and climate change 


Table 3 
Delphi panel scores (lowest score is 1, highest score is 5) for the extent to which sustainability issues are addressed within sustainability assessment tools for bioenergy 
systems. 
Sustainability issues C&I LCA EIA CBA SPA 
Environmental dimension 
Air quality 2 5 4 4 2 
Soil quality 4 2 4 1 1 
Land use 3 3 3 3 5 
Water quality 4 3 4 1 1 
Water use 4 5 4 3 1 
Biodiversity 3 3 3 1 1 
Deforestation 4 3 3 1 3 
Desertification 3 3 3 1 1 
Landscape 2 4 4 2 1 
Agrochemicals 3 3 3 1 3 
Waste 4 4 3 il 1 
Energy balance 4 5 3 5 5 
Fossil resources 3 5 3 4 5 
Social dimension 
Labour conditions 3 1 1 2 1 
Human safety & health 3 1 3 2 1 
Human rights 3 1 1 1 1 
Access to resources 3 1 1 1 1 
Food & energy supply 2 1 1 1 2 
Capacity building 3 1 1 1 1 
Democratic participation 3 1 2 1 1 
Property rights 3 1 1 1 1 
Fair trade 3 1 1 1 1 
Social acceptance 3 1 3 1 1 
Economic dimension 
Viability 3 1 1 4 1 
Long term perspective 3 3 1 3 2 
Reliability of resources 2 3 1 2 2 
Reliability of technology 1 1 1 3 2 
Risk minimisation 2 1 1 1 1 
Local economy 3 1 1 1 1 
Reliability of energy 2 1 1 3 3 
Other desirable developments. 3 1 1 1 1 
Institutional dimension 
Policy and legislation 5 1 5 1 2 
Institutional capacity 2 1 2 1 1 
Institutional well-being 3 1 1 1 1 
Environmental management 3 1 2 1 1 


3924 V. Buytaert et al. / Renewable and Sustainable Energy Reviews 15 (2011) 3918-3933 


issues [24]. Of the six selected tools, social issues are only addressed 
by C&I. The reason behind this is probably the difficulty with 
which social issues of sustainability are translated into measur- 
able quantities, or at least into operational terms. According to 
Becker [1] approaches to quantification and operationalisation of 
social dimensions must be carefully restricted to those issues that 
can be described meaningfully by numerical or analytical tools and 
methods. Yet, C&I cover most of the issues of social sustainabil- 
ity although some sustainability issues such as labour conditions, 
human safety and property rights cannot get a better indicator than 
a rank or go/no go. Only for competition with food, no appropriate 
indicator was found yet in C&I. The institutional dimension is very 
poorly addressed in all of the tools except for the legislative aspect, 
which is of significant importance in EIA and C&I. 

The Kruskal-Wallis analysis that was performed on the results 
of the sustainability issues for each of the sustainability sublevels 
(environmental, social, economic and institutional dimension) 
reinforced the visual conclusions by not revealing any significant 
differences between the tools except for the social dimension (p- 
values for the environmental, social, economic and institutional 
dimension are 0.014, 0, 0.011 and 0.009, respectively). 

To find out which sustainability issues are addressed in EA, sci- 
entific literature was consulted. The relationship between exergy 
and sustainability is well explained in studies of Cornelissen [128], 
Rosen and Dincer [106], Dincer and Rosen [129] and Dewulf et al. 
[130]. It seems that the focus of EA is limited to the environmental 
dimension, in particular emissions and natural resource use. The 
authors state that environmental effects associated with emissions 
and resource depletion can be expressed in terms of one exergy- 
based indicator. In contrast with energy, which can never be lost as 
it is conserved according to the first law of thermodynamics, exergy 
can be lost due to internal irreversibilities. Therefore, sustainabil- 
ity is seen to increase and environmental impact to decrease as the 
exergy efficiency of a process increases [106]. Some studies were 
found relating EA and economics by determining the appropriate 
allocation of economic resources so as to optimize the design and 
operation of a system, or by determining the economic feasibility 
and profitability of a system. Examples of exergy-based economic 
analysis methods are thermoeconomics, second-law costing, cost 
accounting and exergoeconomics. Dewulf et al. [130] give a concise 
overview of the application of EA in the combined environmental 
and economic field, and debate the actual merits of such an exergy- 
based economic approach [101]. No exergy studies were found that 
address social issues of sustainability. 

In general, the major problems in sustainability assessment are 
data availability and methodological agreement. For some sus- 
tainability issues, such as energy balance, methodologies are well 
defined and accepted, but for others, methodologies are heavily 
debated. For example, the ideas on how to inventory and charac- 
terize land use are incomplete and quite diverse [131], not only 
between tools but also within tools. In-depth discussions on how 
to integrate land use within LCA are still going on and can be found 
in [85,132-136] midst others. The same difficulties are faced for 
assessing biodiversity. Although a few indicators related to biodi- 
versity already exist (e.g. species diversity, existence of threatened 
and protected species), there is no agreement on a common denom- 
inator or calculation procedure within sustainability assessment of 
bioenergy systems. At date, the problem is mostly tackled by deter- 
mining protected areas where intervention is not allowed to take 
place. The reason why in most cost-benefit analyses some of the 
environmental and most of the social costs are omitted, is often 
because they are too difficult to estimate from an economic point of 
view and to translate into monetary terms. For the use of agrochem- 
icals, for instance, cost-benefit analyses may consider the costs of 
treating wastewater from the chemical plant where the agrochem- 
icals are produced, but the effect on nature is often omitted because 


it seems too complicated to determine the values for the deterio- 
rated nature. It can be concluded that within existing assessment 
tools the same sustainability issues are often left out of considera- 
tion due to a lack of possible calculation methodologies or available 
data. Future research should focus on those sustainability issues for 
which no common calculation methodology could be defined yet. 


3.2. Tool attributes 


In the second evaluation step, tools were evaluated by the Del- 
phi panel of experts on basis of their tool characteristics regarding 
use of data, followed procedure, reproduction of results, and use of 
the tool. Delphi scores ranged from 1 to 5, corresponding with very 
poor, poor, moderate, good or very good. The resulting data matrix 
can be found in Table 4. Some of the tool attributes are also of impor- 
tance in the next level of comparison (overall model structure) and 
will be discussed in more detail there. 

At first glance, no general differences can be observed 
between the tools. This perception is confirmed by the statisti- 
cal Kruskal-Wallis analysis, which does not detect any significant 
differences between the different tools (p-values for data, proce- 
dure, results and use are 0.453, 0.220, 0.533, 0.192, respectively). 
Although there might not be a statistical significant difference in 
general, many smaller differences can be observed on the various 
aspects in particular. 

Although input data is generally a difficult issue in sustainability 
assessment, LCA shows many advantages. Extensive high-quality 
data sets have been gathered in the past decennium and are reg- 
ularly updated. CBA scores well because of advantages related to 
valuing such as a similar unit for all impacts which facilitates inte- 
gration. Concerning data availability, need and integration, EIA 
shows difficulties. The tool is very site-specific and detailed, which 
requires extensive application of data and hampers integration. 
Low scores are also assigned to SPA, which has the disadvantage 
of being a very recent and not yet fully developed assessment tool. 
Only few sustainability issues are considered and clear assessment 
guidelines are still lacking. This might change in the near future. 

On the aspects of procedure, C&I receives the best overall 
score. Indicators are simple and flexible measures, most often 
quantitative, that represent a state of economic, social and/or envi- 
ronmental development in a defined region. When selected with 
great care, C&I can be a strong measure for sustainability assess- 
ment. But in the case of poorly chosen indicators a variety of 
problems occur, amongst which overaggregation, measuring irrel- 
evant aspects, dependence on a false model, deliberate falsification, 
and incompleteness [137]. Other advantages of C&I are an almost 
full coverage of the sustainability field, whereas other tools rather 
focus on one or two sustainability dimensions, and the ability to use 
static as well as dynamic data, whereas most other tools, except for 
EIA use only static data. A positive aspect to note, is that all tools 
score well on transparency and not any is prone to high subjectivity. 

With regard to the results, all tools generally show high scores. 
Results are in most tools rather compact and clear, easy to under- 
stand without expert knowledge. Only for SPA one weakness can be 
observed for applying a complex calculation method which makes 
it difficult to interpret the results. 

Finally, with respect to the use of the tool, all of them show 
the advantage of being able to integrate stakeholder input into the 
modelling, whereas accessibility forms a bottleneck for each of the 
tools. This means that expert knowledge is needed to apply the 
tool, particularly in the case of EIA and SPA. LCA scores best on 
time consumption and assistance considering the rather limited 
time that is needed for the assessment and the good guidelines and 
assistance that are available, the latter being the opposite for SPA. 
EIA seems to be worst on these two issues, as well when handiness 
is concerned. 
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Table 4 
Delphi panel scores (lowest score is 1, highest score is 5) for tool attributes of sustainability assessment tools for bioenergy systems. 

Tool attribute C&I LCA EIA CBA SPA 

Data 
Availability 3 4 2 3 2 
Need 3 3 2 3 2 
Type 3 5 5 5 5 
Integration 4 1 1 4 1 
Correctness 3 4 3 3 3 

Procedure 
Flexibility 5 1 3 3 3 
Coverage 4 1 1 4 2 
Objectivity 3 3 4 3 3 
Transparency 4 5 4 4 4 
Validation 4 4 2 1 3 
Time scale 4 1 1 5 2 
State-of-the-art 5 5 5 3 2 

Results 
Univocity 5 4 4 3 4 
Simplicity 5 4 4 3 2 
Type 3 5 4 4 5 
Clarity 4 2 3 4 3 
Communication 4 5 3 4 3 

Use 
Flexibility 5 5 5 4 5 
Accessibility 4 3 2 3 2 
Application cost 3 3 3 3 3 
Application field 5 3 4 2 4 
Time consumption 3 4 2 3 3 
Handiness 4 3 2 3 3 
Assistance 3 5 2 3 2 


3.3. Overall model structure 


This part of the evaluation is based on literature review and the 
results can be found in the form of tool profiles in Appendix A. A 
discussion of the results follows here. 

The tool description specifies in short what the overall purpose 
is of using the tool and which aspects are assessed. Although some 
tools are rather used for communication to provide others with 
information (C&I), whereas other tools serve as decision-support 
for operative or strategic decision making (CBA, EIA) or are used 
by companies for internal learning purposes (EA), or for all these 
purposes together (LCA, SPA), they all contribute in some way to 
sustainability assessment and can to some extend be used for the 
assessment of bioenergy systems. 

The tool concept gives an overview of the different methodolog- 
ical steps that must be executed while applying the tool and is 
related to the concept focus of the tool. A tool typically consists of 
a systematic data and/or information gathering procedure and will 
often also include a calculation algorithm [123]. Tools that empha- 
sise procedure are C&I and EIA, whereas the others tools (LCA, 
CBA, SPA and EA) rather focus on the computational algorithm. For 
example, the EIA methodology gives descriptions of an information 
gathering procedure (in order to make decision) in which various 
algorithms may be used, whereas the LCA methodology defines the 
type of computational model to be used (a computational model of 
material and energy flows related to a functional unit). Since pro- 
cedure aims at describing the best way to reach a decision, while 
algorithms aim at finding the better solution, an analytical tool 
might be used as part of a procedural tool [125]. As a consequence 
of their focus on algorithm, analytical tools show the disadvantage 
that sustainability issues that cannot be modelled appropriately 
can hardly be integrated in the assessment. This again is related to 
the rather quantitative or qualitative character of the tool and will 
be discussed further on. 

The object of analysis identifies the focus of decision. If the 
object of analysis is a project, such as a bioenergy implementa- 
tion project, EIA or CBA is the natural choice rather than LCA or 
another method developed for another object, which could be pre- 


ferred when assessing for example 1 MJ of biofuel. C&I as well as 
SPA and EA could lay their focus on any kind of object or system, 
be it a product, process or project. 

Assessment tools usually involve comparison in various ways. 
The basis for comparison is what is being kept constant in a com- 
parison [123] and depends in most cases on the object of analysis. 
For example, in LCA, all analyses are relative to the functional unit, 
which measures the unit good or service that is being delivered. In 
case of bioenergy this would be one unit of the fossil energy ref- 
erence. The same applies for SPA and EA. C&I on the other hand is 
usually not used to make a comparison between different products 
or projects, but rather to make an evaluation of the same project 
over time. EIA does not necessarily require a reference object either, 
however, in most EIA studies, various alternative projects are com- 
pared amongst which a zero alternative [89]. This is also the case 
for CBA. 

The degree of standardisation or harmonisation is an important 
factor because even small differences in methodological factors can 
influence the results. Of all the tools selected for this study, LCA 
is the only tool that is standardised by the ISO. The tool evolved 
over the last three decades from a relatively vague framework 
for conducting assessments, into a rigorous set of internation- 
ally standardised ISO guidelines [71,72,123,138]. However, Dewulf 
and Van Langenhove [109] state that some further methodological 
developments are still needed for the LCIA models and methods. 
For C&I, internationally harmonised frameworks and/or guidelines 
exist for specific systems, often as part of certification systems. 
Also for bioenergy, different C&I frameworks have been developed 
[52-54,121]. According to Moberg [95] an international agree- 
ment on a general procedure exists for EIA and because of legal 
implementation, many countries defined their own harmonised 
procedure. Also for CBA, international guidelines exist. For exam- 
ple, the European Union established an extensive guide for CBA of 
different kinds of investment projects [139]. On the methodology 
of SPA, little information is available because of very recent devel- 
opment of the tool. For EA, no standardisation or harmonisation 
has been performed but quite some information is encountered 
regarding the methodology [104,105,107,110,140]. 
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Similarity can be found between the degree of standardisa- 
tion or harmonisation and the frequency of use. With respect to 
sustainability assessment of bioenergy systems, LCA is one of the 
most well-known and applied tools, together with C&I within cer- 
tification systems. CBA, in its turn, seems to be the most well 
established and utilised tool in economic decision making [95]. EIA 
is a well-known and frequently used tool for site-specific invest- 
ment projects where participation of all the stakeholders involved 
is of great importance. To date, SPA is only used by its developers. 
And although many scientists suggest that the impact of energy 
resource utilization on the environment and the achievement of 
increased resource-utilization efficiency are best addressed by con- 
sidering exergy [105,113,141], the EA tool still is not very frequently 
applied in a non-scientist environment, perhaps due to the unfa- 
miliarity with the exergy concept. 

With regard to the time perspective, C&I is generally seen as a 
retrospective tool, whereas CBA, EIA and SPA have mainly prospec- 
tive purposes, and LCA and EA can be considered as both retro- 
and prospective. Concerning the temporal and spatial scale, these 
aspects are in principle defined by the temporal and spatial dis- 
tribution of the assessed impacts of the system under evaluation. 
Therefore, sustainability indicators within C&I can be divided 
roughly into state and trend indicators and have specific spatial 
scales and time horizons. Becker [1] presents a clear overview 
of sustainability indicators in a space/time frame. In agriculture, 
spatial scales range from plot-level and small-scale gathering to 
agribusiness and large plantations, and temporal scales range from 
short-season crops to perennials and long rotations in swidden 
agriculture [142]. While conventional environmental assessment 
techniques such as EIA focus only on either manufacturing pro- 
cesses or end-of-life disposal (or reuse), LCA considers the life cycle 
of a system, or the entire chain of events and activities that are nec- 
essary to support the product or process [71,138,143]. This is often 
called the cradle-to-grave approach, and has the obvious advan- 
tage of revealing potentially significant but hidden environmental 
impacts. Instead of focussing attention on large, concentrated and 
readily apparent point sources of impacts — for example, a manu- 
facturing plant - LCA also takes into account dispersed activities 
whose cumulative effects may prove to be critical as well [138]. 
In many cases, EA takes on a life-cycle perspective as well, quan- 
tifying the cumulated exergy consumption of a product or process 
from “cradle to grave”. In this regard, it is similar to LCA. Moreover, 
according to Dewulf et al. [101] EA can be part of an LCA, repre- 
senting a method for the life-cycle impact assessment of resource 
consumption. A coupling of Exergy Analysis and life cycle inven- 
tory databases (e.g. Ecoinvent), has been put into practice by the 
authors. The developers of SPA followed another train of thought 
considering geographical system balances of resources and related 
effects rather than a life-cycle approach, what makes the assess- 
ment a snapshot in time rather than an integration of impacts that 
occur over time, and no future developments are taken into con- 
sideration. A cost-benefit analysis can give useful results for the 
present costs and benefits, but future benefits are strongly depen- 
dent on the interest rate applied in the analysis. Jorgensen [103] 
state that the use of depreciation of future benefits due to introduc- 
tion of an interest rate will inevitably emphasize too much present 
benefits. In addition, predictions on future interest rate have a very 
high uncertainty. 

The spatial scale is related to the spatial boundaries and the spa- 
tial horizon of the analysed system. Whereas most of the selected 
tools can be applied on different spatial scales, EIA is a particu- 
larly site-specific tool [125]. Although all the tools make use of 
environmental data, the environmental data of EIA are usually 
more detailed and site or project specific. For example, EIA can 
often involve taking into account the duration and concentration 
of emitted pollutants and the assessment of actual impacts on the 


environment [123], whereas the other tools keep it less situation- 
specific to serve as rather generic tools. The site-specificity of EIA 
makes it a time and resource demanding tool. On the opposite side, 
indicators, and in particular social indicators, can be used to refer to 
all levels of the spatial hierarchy, with varying degrees of relevance. 
Yet, in different societies with varying cultural and legal practice, 
assessment of social indicators will always be a complex issue that 
cannot easily be put into operation [1]. 

There is no straightforward answer whether an assessment tool 
should be purely quantitative or both quantitative and qualitative. 
Not all sustainability issues can easily be quantified. The accuracy 
of the outcome strongly depends on the accuracy of the measure- 
ments and the modelling within the tool. Besides, the combination 
of qualitative and quantitative results impedes aggregation into one 
single outcome. Therefore, in some tools only well-defined quan- 
titative parameters are counted in. On the other hand, in order to 
fully address sustainability, all issues should be considered, also the 
non-quantifiable ones. Of all the tools regarded in this study, only 
C&I and EIA have both a quantitative and qualitative character. The 
other tools are purely quantitative. 

The quantitative and/or qualitative character of a tool also influ- 
ences the way in which the results are reproduced. According to 
Becker [1], three basic approaches can be distinguished: unranked 
lists of heterogeneous indicators, scoring systems with unified 
dimensions, and system descriptors. C&I, LCA, EIA and SPA belong 
to this first group, although results might be partially aggregated for 
example into impact categories as is the case in LCA. The advantage 
of listing results as heterogeneous indicators gives the advantage 
of transparency, but the disadvantage of complex interpretation. 
Because of their ability to transform all parameters into one unit, 
CBA and EA are tools that belong to the second group of scoring sys- 
tems with unified dimensions. Such tools have the advantage that 
they permit comparison of and calculation with different quantities 
in a uniform dimension. However, according to Becker this assumes 
a congruent basis of calculation, which in reality differs consider- 
ably depending on the underlying value judgement. Furthermore, 
in case of CBA monetary values are not sufficiently consistent with 
ecosystem structure and function, and therefore their aggregation 
may lead to inadequate environmental decisions. 

The assignment of weight scores is considered as a very contro- 
versial step mainly because it involves social political and ethical 
value choices [109]. Therefore, weighing of impacts is in most tools 
not integrated, or indicated as an optional step. For example, in 
the inventory phase (LCIA) of LCA, a number of methods is pro- 
posed to conduct weighing. These methods are divided in two 
general groups: (1) problem-oriented approaches and (2) damage- 
oriented methods. Examples of the problem-oriented approaches 
are the CML-method and EDIP method, and examples of damage- 
oriented methods are EPS 2000 and Eco-Indicator 99. The IMPACT 
2002+ method attempts to link the problem- and damage-oriented 
approaches in a common framework [109]. 

An optimal application of a model or tool requires the execution 
of a sensitivity and uncertainty analysis before the final conclusions 
and recommendations of the study are made. However, because 
of time and/or resource restriction, this part is often omitted. Of 
the six tools considered in this study, only two of them, LCA and 
CBA, include a sensitivity analysis and to a limited extend also 
an uncertainty analysis in their methodology, however optional. 
Only in a few case studies sensitivity and/or uncertainty analyses 
were conducted although it was often stated that the methodol- 
ogy applied was prone to considerable uncertainties. With regard 
to assessment tools in general, three kinds of uncertainty can be 
distinguished: data uncertainty, model uncertainty and scenario 
uncertainty. Small-scale models are less prone to uncertainty than 
large-scale models, because the latter cover in most cases longer 
time spans, and therefore need to make more probability assump- 
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Fig. 2. Possible field of application of sustainability assessment tools based on their 
temporal focus, along with the object of focus of the tool. 


Adapted from Ness et al. [15]. 


tions [1]. More specific, in case of C&I uncertainties can be caused by 
the diversity of the set of criteria and indicators, criteria content, 
indicator quality, lack of validity, poor data sets amongst others 
[144]. In life-cycle assessment, various aspects such as input param- 
eter values, system boundaries, allocation procedure, and fossil 
reference system, give rise to wide ranges of results [83]. In a cost- 
benefit analysis the accuracy of the outcome highly depends on 
which costs are included in the analysis and how accurately costs 
and benefits have been estimated. Various peer-reviewed studies 
[145,146] show that differences between actual costs and bene- 
fits and estimated costs and benefits can be up till more than fifty 
percent and that the outcomes of cost-benefit analyses should be 
treated with caution because they may be highly inaccurate. Fur- 
thermore, Flyvbjerg et al. [147] state that inaccurate cost-benefit 
analyses may be argued to bea substantial risk in planning, because 
inaccuracies of the size documented are likely to lead to ineffi- 
cient decisions. EIA is prone to similar uncertainties as in C&I in 
the phases in which indicators are used and to some extend to the 
same uncertainties of CBA in case a financial analyses is executed. 
For SPA, no information was found on the risk of uncertainty while 
applying the tool. In EA at last, a certain degree of uncertainties is 
found in exergy estimations of products and processes. 


3.4. Application of tools 


In the fourth and last level of the evaluation, an approach was 
followed in which two factors are considered: temporal char- 
acteristics and focus area. As is discussed earlier in this paper, 
temporal characteristics evaluate the time frame wherein a product 
or projects has an impact, as well as the time perspective applied 
to the object analysed which can be retrospective or prospective. 
The former is accounting/monitoring, keeping record of progress, 
which can be useful when searching for preferable changes in for 
example production, behaviour or policies and to indicate sustain- 
able development. Prospective approaches, on the other hand, are 
seeking to predict future situations and can thereby facilitate com- 
parison between different new products, plans, etc. [123]. The focus 
areas can be (1) indicators and indices, which can be further bro- 
ken down into non-integrated and integrated, (2) product-related 
assessment tools with the focus on the material flow and/or energy 
flows of a product or service from a life cycle perspective, and (3) 
integrated assessment, which are tools usually focused on policy 
change or project implementation. Fig. 2 gives an overview for the 
six tools regarded in this paper. As stated by Ness et al. [15] indica- 
tors and indices (i.e. aggregated indicators), which are continuously 
measured and calculated, allow for the tracking of longer-term 
sustainability trends from a retrospective point of view. Product- 
related tools such as LCA and EA allow both retrospective and 
prospective assessments. Integrated assessment tools like EIA, CBA 
and SPA are mostly used for supporting decisions related to a policy 
or a project in a specific region and have an ex-ante focus. Many of 
these tools are based on systems analysis approaches and integrate 
environmental and society issues. 

Besides the area of applicability, a categorization of tools can also 
be made concerning their development and frequency of use. For 


LCA, CBA and EIA, there are relatively well established, transparent 
and generally agreed guidelines available, whereas for C&I and for 
EA the practitioner is free to use a personal interpretation of the 
general principles. Newer tools such as SPA represent an area where 
guideline establishment is still in early developmental stages. 


3.5. Integration of existing tools 


As can be concluded from the previous analysis, the various 
existing sustainability assessment tools are not necessarily com- 
plementary, but some of them may be well combined in practice 
in the form of a toolbox. For the assessment of bioenergy systems, 
impacts must be considered both in short and long term and it 
involves tracking on-going activities (the accounting approach), as 
well as choosing between alternative materials/products/actions 
(the project approach). The former is important for the generation 
of alternatives, the latter for the comparison of alternatives. There- 
fore, tools that focus on the former (i.e. analytical tools such as LCA, 
CBA, SPA and EA) might be used as part of a tool that focuses on 
the latter (i.e. procedural tools such as C&I and EIA). As the pro- 
cedure of C&I seems to be more clear, transparent and efficient 
compared to this of EIA, it would be recommended to use the pro- 
cedural framework of C&I within the toolbox. Nevertheless, EIA also 
has some very useful procedural parts such as the screening process 
and mitigation of negative impacts, which could be integrated into 
the C&I framework. With regard to the analytical part of the tool- 
box, the LCA methodology stands out for its well-developed and 
clear assessment of environmental impacts, although EA could bea 
decent alternative for the calculation of energy efficiencies. Where 
economic sustainability indicators are concerned, CBA seems to be 
the most useful tool to provide calculation algorithms, although 
many issues in this sustainability dimension for which no common 
calculation algorithm exists yet will require considerable additional 
research efforts. In the social sustainability field, the most useful 
calculation procedures are encountered in the C&I methodology, 
however also in this field commonly accepted calculation proce- 
dures are still lacking for various sustainability issues. 

Even more interesting would be the construction of a combined 
software toolbox, which includes an application of geographic 
information systems (GIS) and remote sensing for determining 
spatial characteristics, together with a model that assesses the tem- 
poral dimension of impacts, as well as a methodology to conduct 
sensitivity and uncertainty analysis, and finally also an instrument 
which enables decision makers to weight sustainability issues and 
compare alternatives against each other. It is clear that, for such 
a tool to be created, far more research has to be conducted. Sci- 
entists of many different fields of expertise have to combine their 
knowledge and forces. Solutions must be found for those sustain- 
ability issues for which no methods can be recommended yet. 
Besides, this paper represents only a small selection of sustainabil- 
ity assessment tools. It would be recommendable to extend this 
study to other assessment methodologies such as Multi-Criteria 
Analysis, Risk Assessment, Strategic Environmental Assessment, 
Social Impact Assessment, Ecological Footprint, Life Cycle Costing, 
and many more. 


4. Conclusions 


The use of biomass for energy as part of a sustainable devel- 
opment strategy is heavily debated due to many concerns about 
sustainability. Bioenergy systems are complex systems having 
an impact on various levels of nature and society and involving 
many stakeholders. Furthermore, sustainability is an essentially 
contested notion because it is inherently complex, normative, 
subjective and ambiguous [18]. As a consequence, sustainability 
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assessment of bioenergy systems is a highly demanding and com- 
plicated task. Nevertheless, many researchers have been engaged 
in a quest to develop tools and methods to tackle this difficult job. 
While many started working from a mono-disciplinary basis, others 
made an attempt to develop generic tools and instruments. This has 
resulted in a diverse field of approaches [23]. This paper tried to give 
a better insight in the complex world of sustainability assessment 
of bioenergy systems by analysing and comparing the applicabil- 
ity and performance of a number of basic sustainability assessment 
tools that can be applied in this sense. First, a framework was con- 
structed for the evaluation of the selected assessment tools, which 
takes all issues into account that are of importance for assessing 
bioenergy. In the second step a Delphi panel of experts was con- 
sulted, as well as a considerable amount of literature. In the third 
step, results were analysed and shaped in comprehensible forms. 
It can be concluded that not any of the selected assessment tools 
is judged as completely satisfying on all aspects of the evaluation 
framework. Although some tools are already very well developed, 
they all have severe shortcomings for sustainability assessment of 
bioenergy, whether it is from a theoretical point of view and/or from 
the practical side. Especially the absence of appropriate indicators 
and calculations of a number of sustainability issues, as well as the 
lack of data and uncertainty analysis, are often problematic. For 
the moment, not any of the selected tools could be recommended 
to perform an overall and integrated sustainability assessment of 
bioenergy systems. It appears that, in the first place, a good defini- 
tion of the sustainability problem is crucial. A different assessment 
focus may lead to a different choice of assessment tool. Further- 
more, all tools display particular advantages and disadvantages, 
which means that trade-offs are inevitable and a balance must be 
found between scientific accuracy and pragmatic decision making. 
LCA, for instance, performs undoubtedly better on the environmen- 
tal part of sustainability while CBA is a better choice with respect to 
assessing economic issues. EIA shows quite some resemblance to 
LCA in terms of basic equations, and addresses even more sustain- 
ability issues, but the site-specific character of the tool makes it very 
time and resource demanding. C&l is probably the most practical 
and frequently used tool, but as is the case for EIA, the assessment 
results in a list of parameters with varying units which makes it dif- 
ficult to interpret. EA seems to be a worthy alternative for LCA but 
the tool appears to be rather difficult to interpret and is so far not 
often practised for sustainability assessment. CBA in turn is praised 
for its simple results and its suitability for comparison, but mon- 
etisation is a source of subjectivity and uncertainty, and reduces 
transparency. SPA for its part follows an interesting approach that 
is completely different from the others looking at perturbations ina 
system instead of a products life cycle, but the early developmental 
stage of the tool impedes a generic use. 

In addition, it can be concluded that the various existing assess- 
ment tools are not necessarily complementary, but might be com- 
bined in practice in the form of a toolbox. It would thus be interest- 
ing to investigate the possibility of creating a toolbox that combines 
the clear and transparent procedure of C&I with some interesting 
procedural parts of EIA such as screening and mitigation, supple- 
mented with calculation algorithms of LCA and CBA for respectively 
environmental and economic sustainability indicators. Therefore, 
a more comprehensive interdisciplinary approach is necessary to 
align the different tool characteristics and focuses, together with 
additional research on those sustainability indicators for which no 
common calculation procedure could be developed yet. 
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Table A1 
Tool profile of Criteria & Indicators (C&I). 


General 

Tool description 
Criteria and Indicators (C&I) are a policy and management tool used to 
define, assess and monitor progress towards sustainable management in 
a given area, over a period of time taking into consideration the 
environmental, social, economic, cultural and spiritual needs of the full 
range of stakeholder groups in the areas concerned [49]. 

Tool concept 
After the definition of goal and scope, a conceptual framework is 
developed that is generic and broadly applicable. The framework is 
divided into principles, criteria, indicators and metrics and/or verifiers. 
For those issues for which no indicators are currently available, 
reporting protocols can be formulated. Eventually, indicators are 
assessed and verified. 
Criteria define the essential elements against which sustainability is 
assessed. Each criterion relates to a key element of sustainability, and 
may be described by one or more indicators. Indicators are parameters 
which can be measured and correspond to a particular criterion. They 
measure and help monitor the current condition and change over time 
in quantitative, qualitative and descriptive terms [49]. Metrics and 
verifiers provide additional data or information that enhances the 
specificity of the ease of assessment of an indicator. 

Object of analysis 
C&I can be used to assess all kinds of systems on different levels, 
although higher levels are less frequently used (national, supranational, 
global). For example, in the case of agricultural land use systems, the 
system under consideration can be the cropping system (plot level), 
farming system (farm level), watershed/village (local level), 
landscape/district (regional level) or a hierarchy of different levels can 
be used [1]. 

Type of comparison (between alternatives, within a studied system, 
against a reference) 
C&I is mostly used in an absolute way (no comparison of different 
systems or scenarios), although C&I may also be used to monitor a 
system over time. 

Standardisation/harmonisation 
There is no standardisation, but various initiatives have tried to develop 
internationally harmonised C&I frameworks, for example in sustainable 
forest management or sustainable palm oil production [50,60]. 

Frequency of use 
C&I is a very frequently used tool because of its ease of use, flexibility 
and transparency. The International Institute for Sustainable 
Development (IISD) provides an overview of currently about 600 
indicator initiatives including the type of initiative, the nature of public 
involvement, geographic scope and project goals [148,149]. 

Methodological 

Time frame (retro-/prospective and temporal boundaries) 
As a monitoring tool, C&I is retrospective, which can be useful to search 
for preferable changes and to indicate sustainability. 
On indicator level, a difference can be made between trend indicators 
and state indicators. Trend indicators capture the dynamic aspect of 
sustainability over time (e.g. yield trends, depletion rate) whereas state 
indicators reflect the condition of the respective (eco)system [1]. 

Spatial scale (spatial boundaries) 
C&I can be applied at different levels: management level, regional, 
national [49]. 
Also on indicator level, there are multiple spatial scales depending on the 
indicator used [54]. An example on small spatial scale is the yield trend 
and on large spatial scale is the discount rate of resource depletion [1]. 

Procedure/algorithm focus 
Procedure focus. 

Quantitative/qualitative 
Both quantitative and qualitative. 

Results (listed or aggregated) 
Results are listed with different units, but an indicator might be an 
aggregation of different effects (e.g. GHG balance). 
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Table A1 (Continued) 


Table A2 (Continued) 


Weight scores 
There is no weighing methodology included in C&I. In some cases, Multi 
Criteria Analysis (MCA) is used to weight and aggregate results. 
Sensitivity & uncertainty 
There is no sensitivity nor uncertainty analysis included in C&I although 
the tool is subject to many uncertainties caused by the diversity of the 
set of criteria and indicators, criteria content, indicator quality, lack of 
validity, poor data sets amongst others [144]. 


Table A2 


Tool profile of Life Cycle Assessment (LCA). 


General 

Tool description 
Life Cycle Assessment (LCA) assesses, in a systematic way, the environmental 
issues and impacts of product systems, from raw material acquisition to final 
disposal, in accordance with the stated goal and scope [71]. 
In a bioenergy perspective, LCA is mostly used to conduct an energy and GHG 
balance. 

Tool concept 
There is no single method for conducting LCA. Organizations have the 
flexibility to implement LCA as established in the International Standard, in 
accordance with the intended application and the requirements of the 
organization [72]. 
The ISO standard defines 4 steps: (1) Goal and scope definition (defined by ISO 
14041 [150]); (2) Inventory analysis (LCI, defined by ISO 14041 [150]); (3) 
Impact assessment including classification, characterisation and valuation 
(LCIA, defined by ISO 14042 [151]); (4) Interpretation (defined by ISO 14043 
[152]) or improvement assessment [117]. 

Object of analysis 
Products and services. 

Type of comparison (between alternatives, within a studied system, against a 
reference) 
LCA is a relative approach, which is structured around a functional unit. This 
functional unit defines what is being studied. All subsequent analyses are then 
relative to that functional unit, as all inputs and outputs in the LCI and 
consequently the LCIA profile are related to the functional unit. 

Standardisation/harmonisation 
The LCA methodology is standardized by the ISO 14040-14043 [71,150-152]. 

Frequency of use 
LCA is one of the most frequently used systems analysis tools. 

Methodological 

Time frame (retro-/prospective and temporal boundaries) 
LCA may be used prospectively when used for development of new products, 
strategies, etc. and retrospectively when used to improve processes and 
products, for purchasing decisions, etc. [95]. 
LCA takes the whole lifetime of a function in account. The depth of detail and 
time frame of an LCA may vary to a large extent, depending on the goal and 
scope definition. 
Time is an important factor in calculations of the annual life-cycle 
performance of biofuels. Both energy inputs and outputs and emissions occur 
at different points in time. For example emissions from land use due to 
changes in aboveground biomass occur mainly during the short period when 
the land is converted for cultivation. The chosen time scale for amortization of 
these emissions can change the results drastically. The IPCC uses a default 
value of 20 years while Greenpeace aims for a 10-year period [82]. 

Spatial scale (spatial boundaries) 
Since the assessment is not site-specific, there are no geographical boundaries 
[95]. 
LCA can be used by governments, private firms, consumer organisations, and 
environmental groups as a decision support tool. The scope of the decisions 
covered by LCA ranges from broad management and policy choices to specific 
selection of product or process characteristics during design [138]. 

Procedure/algorithm focus 
Algorithm. 

Quantitative/qualitative 
Quantitative. 

Results (listed or aggregated) 
In the LCIA phase, LCI results are assigned to impact categories. For each 
impact category, a life cycle impact category indicator is selected and the 
category indicator result (indicator result) is calculated. It is this list of 
indicator results (LCIA results or LCIA profile) that provides information on the 
environmental issues associated with the inputs and outputs of the product 
system [71]. 


Weight scores 
Weighing of impacts is an optional step in the LCIA phase. A number of 
methods exist, divided in two general groups: (1) problem-oriented 
approaches and (2) damage-oriented methods. Examples of the 
problem-oriented approaches are the CML-method and EDIP method. 
Examples of damage-oriented methods are EPS 2000 and Eco-Indicator 99. 
The IMPACT 2002+ method attempts to link the problem- and 
damage-oriented approaches in a common framework [109]. 
Using the ISO standard, the valuation step is not allowed for external studies 
since there is no scientific basis for reducing LCA results to a single overall 
score or number because weighing requires value choices. 

Sensitivity & uncertainty 
Sensitivity analysis is part of the fourth LCA stage in ISO [71]. 
Although various aspects such as input parameter values, system boundaries, 
allocation procedure, and fossil reference system, give rise to wide ranges of 
results [83], no uncertainty analysis is included in the ISO. 


Table A3 
Tool profile of Environmental Impact Assessment (EIA). 


General 

Tool description 
Environmental Impact Assessment (EIA) is used to evaluate potential 
environmental impacts - considering natural, social and economic issues — 
that a proposed project may have, with the aim to reduce the negative 
effects [89]. 

Tool concept 
More than 100 countries worldwide have EIA systems. They vary greatly in 
terms of procedure and practice. The scope and methodologies of EIA have 
evolved greatly over the past decade. Advances include consideration of 
biodiversity and climate change issues; increasing applications to policy, 
plans and other strategic decisions; and review of trade, privatisation and 
structural adjustment initiatives [24]. 
Although the amount of steps in an EIA process may differ, in general the 
following steps can be distinguished: (1) Screening; (2) Scoping; (3) Impact 
assessment; (4) Mitigation; (5) Reporting (EIS); (6) Reviewing; (7) 
Decision-making; (8) Implementation [24]. 
EIA is very site-specific. Most countries have lists of activities for which EIAs 
are required (e.g. mining or major construction works). In addition, some 
countries have identified sensitive environments (e.g. estuaries of cultural 
heritage sites) for which EIAs are needed [24]. 
Stakeholder participation is an important issue in EIA. 

Object of analysis 
A policy, plan, programme or (mainly) a project. 

Type of comparison (between alternatives, within a studied system, against a 
reference) 
Comparison is based on the project or site being analysed. There is no 
obvious need for a reference object, since the purpose of the tool is to 
describe the effects a planned action will have. However, EIA should 
consider alternatives to the proposed project localisation and design, 
including the zero alternative [95]. 

Standardisation/harmonisation 
There is an international agreement on general procedure [95]. Often 
countries have their own procedure. 

Frequency of use 
EIA is a well-known and frequently used tool. EIAs are performed in several 
countries including some developing countries [95]. 

Methodological 

Time frame (retro-/prospective and temporal boundaries) 
Prospective, foresees impacts of planned activities and improves long-term 
viability of projects [95]. 
For a major project, an EIA may take considerable time, manpower and 
resources. The first 2 stages are very important to determine the required 
extent and focus of the EIA [153]. 

Spatial scale (spatial boundaries) 
(Same as time frame) The spatial boundaries are defined by the spatial 
distribution of important direct and indirect impacts caused by the proposed 
project or plan. 

Procedure/algorithm focus 
Procedure focus. 

Quantitative/qualitative 
Both quantitative and qualitative. 

Results (listed or aggregated) 
Listed with several different units. EIA reports tend to present impacts in 
separate chapters for environmental, social and economic aspects [153]. 
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Table A5 (Continued) 


Weight scores 
No available information on the use of weight scores. 
Sensitivity & uncertainty 
No sensitivity or uncertainty analysis is included in the EIA procedure. 


Table A4 


Tool profile of Cost Benefit Analysis (CBA). 


General 

Tool description 
Cost Benefit Analysis (CBA) is a method trying to estimate the total impact of 
a project on society by calculating social costs and benefits. Environmental 
impacts are included by valuation of these, converting them into monetary 
terms [94]. 

Tool concept 
Steps: (1) identification of problem and alternative solutions; (2) 
identification of social costs and benefits for each alternative; (3) valuation 
of costs and benefits (usually in monetary terms); (4) allocation of costs and 
benefits over the project time; (5) calculation of net present value (NPV); (6) 
ranking alternatives by NPV; (7) sensitivity analysis should be performed; 
(8) presenting recommendation [94]. 

Object of analysis 
Projects (or sometimes decisions on higher strategic levels e.g. programmes) 
[95]. 

Type of comparison (between alternatives, within a studied system, against a 
reference) 
The zero alternative is used as reference and different alternatives should be 
compared [95]. 

Standardisation/harmonisation 
CBA follows an established methodology. Rough guidelines are available 
[94]. 

Frequency of use 
CBA is the most well established and utilised tool in economic decisions 
situations [95]. 

Methodological 

Time frame (retro-/prospective and temporal boundaries) 
Prospective: estimation of future costs and benefits. 
Future generations should be considered, which implies no theoretical time 
boundary [95]. Preference of time is considered through discounting, which 
is a way of including the change of capital over time. Discounting is done by 
deciding on a discount rate defining present values of future costs/benefits. 

Spatial scale (spatial boundaries) 
Economic and geographical boundaries are set according to the 
development considered in the CBA study. CBA is a partial economic analysis 
with no feedback from other markets or regions [95]. 

Procedure/algorithm focus 
Algorithm focus. 

Quantitative/qualitative 
Quantitative. 

Results (listed or aggregated) 
Results are in monetary unit. Costs and benefits that are impossible to 
calculate in monetary terms may be presented as intangibles [95]. 
Sometimes Multi-criteria Analysis (MCA) or Social Impact Analysis (SIA) is 
used to count for the intangible costs and benefits in the final result. 

Weight scores 
Valuation of costs and benefits or the use of MCA or SIA may involve some 
kind of weighing. 

Sensitivity & uncertainty 
A sensitivity analysis is included in the procedure of the tool (step 7) but not 
necessarily performed. 
CBA is subject to huge uncertainty because of many valuations. Since the 
result is presented in an aggregated form, this uncertainty may often be 
forgotten [95]. 


Table A5 


Tool profile of Exergy Analysis (EA). 


General 

Tool description 
Exergy Analysis (EA) is a method that uses the conservation of mass 
and conservation of energy principles together with the second law 
of thermodynamics for the analysis, design and improvement of 
energy and other systems. The exergy of an energy form or a 
substance is a measure of its usefulness or quality or potential to 
cause change. The exergy method is a useful tool for furthering the 
goal of more efficient energy-resource use, as it enables the 
locations, types and magnitudes of wastes and losses to be identified 
and meaningful efficiencies to be determined [129]. 


Tool concept 
Exergy can be accounted for both in energy and material. It can 
describe loss of natural resources in physical terms [154]. Due to 
entropy generation, the energy that can be made available from the 
outputs (exergy embodied in outputs) is less than the energy that 
can be made available from the inputs (exergy embodied in inputs), 
although the total energy of the outputs equals the total energy of 
the inputs. This quality degradation takes place in all physical and 
chemical processes, whether they take place in the natural 
ecosystem (production of biomass, . . .) or in the industrial ecosystem 
(production, consumption, ...) and it is quantifiable by the loss of 
exergy. For sustainability assessment of products and production 
pathways, three interacting sections are taken into account: the 
production and consumption chain, processes within the natural 
ecosystem and processes in the industrial metabolism [109]. 
Steps: (1) estimation of energy input to the system; (2) calculation of 
exergy input (each energy contribution is multiplied by its energy 
quality (exergy) factor); (3) estimation of energy output from the 
system; (4) calculation of exergy output (as 2); (5) output is divided 
with input and the exergy ratio is obtained [154]. 

Object of analysis 
Products, projects and also economies [95]. 

Type of comparison (between alternatives, within a studied system, 
against a reference) 
The results of EAs are always relative to the specified reference 
environment, which in most applications is modelled after the 
actual local environment [106]. An EA study can serve to obtain 
absolute exergy values of products and processes as well as to 
compare alternatives to find the more exergy efficient one [95,101]. 

Standardisation/harmonisation 
No harmonisation of standardisation. 

Frequency of use 
To some extent used in a rather expertised environment. Mostly 
used for optimisation of energy processes. In recent literature, the 
exergy concept has been shown to be a powerful tool in the 
sustainability assessment of technology and in the analysis of 
bioenergy systems [102]. 

Methodological 

Time frame (retro-/prospective and temporal boundaries) 
Prospectively for estimating the efficiency of potential developments 
and retrospectively for searching potential beneficiary changes [95]. 
No temporal boundaries. In many cases, Exergy Analysis methods 
take on a life-cycle perspective, quantifying the cumulated exergy 
consumption of a product or process from “cradle to grave” [101]. 

Spatial scale (spatial boundaries) 
Geographical boundaries are stated according to the object under 
study [95]. 

Procedure/algorithm focus 
Algorithm focus. 

Quantitative/qualitative 
Quantitative. 

Results (listed or aggregated) 
Results are aggregated and expressed in Joule of exergy. 

Weight scores 
No need for weighing factors by using one single unit that is 
physically interpretable and stable with time [101]. 

Sensitivity & uncertainty 
Sensitivity and uncertainty analyses of EA results were performed by 
Dewulf et al. [101,155]. 


Table A6 
Tool profile of System Perturbation Analysis (SPA). 


General 

Tool description 
System Perturbation Analysis (SPA) identifies the best usage of limited 
resources such as land, wood waste or imports, in terms of fossil energy 
savings or GHG emissions within a given system. 

Tool concept 
A system is defined in which the possible scenarios take place. 
Conversion routes (resource-conversion-product), utilities, and targets 
are taken into account. 
Each SPA scenario is actually a set of perturbations of resources and 
targets. For each perturbation impacts are calculated. 

Object of analysis 
Products, projects and economies. 
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Table A6 (Continued) 


Type of comparison (between alternatives, within a studied system, 
against a reference) 
In an SPA study, alternatives are compared to a reference scenario 
(actual scenario or zero alternative). 

Standardisation/harmonisation 
No standardisation and very little information on the SPA methodology 
is available. 

Frequency of use 
SPA is recently developed and to date only used by the developers 
[115,116]. 

Methodological 

Time frame (retro-/prospective and temporal boundaries) 
Prospective, foresees impacts of possible projects, products or processes. 
SPA takes the whole lifetime of a function into account. 

Spatial scale (spatial boundaries) 
SPA considers a given system where resources are transformed into 
products through a given conversion route. Examples exist already for 
regional or national systems [115] and for single processes [116]. 

Procedure/algorithm focus 
Algorithm focus. 

Quantitative/qualitative 
Quantitative. 

Results (listed or aggregated) 
Listed, expressed in several different units. 

Weight scores 
No use of weight scores. 

Sensitivity & uncertainty 
No information was found on any sensitivity or uncertainty calculations, 
but in the study of Delattin et al. [116] an error calculation was made. 
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